Around 2,000 farmers from the island of Crete, most of them carrying their characteristic wooden canes, gathered in front of the agriculture ministry after arriving by ferries.
They began pelting riot police with tomatoes and other items, broke windows in the ministry and set fire to dustbins.
Police, who blocked their route, responded with tear-gas, and arrested four people.
“The first floor of the building sustained damage, it is fortunate that no staff were hurt," agriculture minister Evangelos Apostolou later told reporters.
Apostolou called on the leaders to contain “extreme" elements in their midst, as officials said that far-right militants had joined parts of the protest.
The junior interior minister for police, Nikos Toskas, said 10 police officers had been hurt, two of them requiring hospitalisation.
“The plan is for farm vehicles to be parked aside so the farmers can participate in the protest without causing confusion in the city," Toskas told state TV ERT.
“We will safeguard the city centre and the protests," he said.
Including the Cretans, farmers’ groups were descending on Athens from at least three sides.
Police also confronted farmers from the Peloponnese peninsula who tried to break through barriers into the city centre from a western suburb with their farm vehicles.
Police also fired tear-gas there after a hooded protester smashed the windshield of a squad car.
“Far-right elements are gathered in this area, trying to create confusion," Toskas told Skai TV.
Another large group from the north plans to camp on central Syntagma Square, opposite parliament, until Sunday.
The farmers are angry about government plans to increase their social security contributions as part of a wider reform of the country’s ailing pension system. They also reject plans to double their income tax by 2017 and scrap benefits such as cheaper fuel.
The government has banned the farmers from using tractors in the demonstration, but a deal was reached to allow a symbolic procession of 17 of the vehicles in the capital.
Since mid-January, farmers have used their tractors to block dozens of highways.
Earlier this month they also began blockading border crossings to Bulgaria and Turkey.
But following an appeal from Sofia on Tuesday, they allowed trucks to pass for several hours a day at one crossing into Bulgaria.
The farmers threatened to further escalate the protest by blocking ports and airports at the weekend.
“We cannot be competitive with a tax rate of 23% when elsewhere in Europe it’s 3% or zero," one farmer on the western Athens roadblock said.
Another farmer raised the longstanding issue of low prices facing producers who are at the mercy of middlemen.
“We sell our oranges at 10 cents a kilo and on the market they cost 50-60 cents," he said.
Prime Minister Alexis Tsipras, whose options are limited after signing a new bailout with Greece’s international creditors in August, offered the farmers a compromise on Wednesday.
“We are open to a substantial, honest dialogue with the farmers," Tsipras said in televised comments to his cabinet.
“There is significant room for improvement on their social security contributions, on the issue of when the measures take effect, and generally over the need to protect their income," he said.
Greece’s creditors the European Union and the IMF have insisted on the reforms as a condition for loans to the debt-ridden country.
The country’s statistical office Elstat meanwhile reported that gross domestic product (GDP) fell back 0.6% in the last quarter compared with the previous three-monthly benchmark.
On a year-on-year basis, GDP retreated by 1.9%.