The incident occurred late on Friday night after the cows were given their feed at the cow shed run by non-governmental organisation Kothuru Tadepalli.
While 98 cows died hours later, the remaining are battling for life, police said.
Sahu, a cow shed committee member, said the cows started collapsing a couple of hours after they were given the feed.
Officials suspect food poisoning as the cause of the death. Police summoned veterinary doctors to conduct an autopsy on the dead cows.
Differences between the cow shed management is believed to have led to the incident as one group suspects that the feed was poisoned.
Krishna District Collector Mohammed Imtiaz visited the gaushala and ordered a probe. He said the death of a large number of cows was unfortunate.
The cow shed had more than 1,000 cows, most rescued from slaughter houses. Some cows were donated by people.
Police said they were questioning the employees who fed the animals and were also checking CCTV footage to find out if any outsider entered the cow shed before the incident.
As many as 24 cows had died at the cow shed run by the same committee near Kanaka Durga temple in Vijayawada two years ago. The cows were fed with stale wheat flour that resulted in their falling sick and ultimately their death. After the incident, the committee shifted the cow shed to the city outskirts.
( With inputs from IANS )