These drivers are for different use

• via_agp is needed to get agp working
• via is needed to get drm working.

If only via is loaded, drm will work, but will fail to acquire AGP, and instead use only the PCI path which for the driver you are trying to use is available only for root since it tries to access registers not even available for an insecure dri client.