Silvestras Gimžauskas was born on 13 October 1845 in Kirdeikiai, in Linkmenys valsčius (a territorial administrative unit). He attended primary school in Linkmenys and went to gymnasiums (secondary schools) in Švenčionys and Daugavpils (then known as Dünaburg). Having completed his studies at the Vilnius Seminary, he was ordained a priest in 1876. He continued his studies in 1877–1878 at the Saint Petersburg Roman Catholic Theological Academy, but due to illness he did not complete his studies there.
Father Gimžauskas ministered to the pastoral needs of a parish in Grodno Governorate, and in Žiežmariai, Vidiškės, Kietaviškės, Valkininkai, Giedraičiai and Bagaslaviškis. His achievements in the Aukštaitija region have been compared to those of Bishop Motiejus Valančius in Samogitia.
He was an active educator, collected folklore and information about Lithuanian language and ethnography, published Lithuanian books, took an active part in the distribution of Lithuanian-language publications (prohibited in the Russian Empire at that time), and played an active role in the temperance movement. He cooperated with various periodicals, including the newspapers Aušra, Šviesa and Vienybė lietuvninkų and the magazine Žemaičių ir Lietuvos apžvalga. He translated works into Lithuanian, including Joseph Haydn's oratorio The Creation, Robert Führer's Mass and Wolfgang Amadeus Mozart's Te Deum. He prepared a primer (introductory textbook) to teach the fundamentals of reading and writing (1888). His poetic work consists of the collections Linkmenės (1870), Ant Naujų metų Lietuvai dovanėlė (1879), Lietuvos Bičiuolis (1881), and the two books Jokaunos dainos (1892–1893).
He prepared the first patriotic Lithuanian proclamation in 1890, printed in the USA, in which he urged the residents of Lithuania Minor (Prussian Lithuania) not to send their children to non-Lithuanian schools, and to cherish and defend the Lithuanian language. He was a prominent advocate of Lithuanian identity in the Vilnius region. He was especially active in Valkininkai.
In 1891, the Rev. Silvestras Gimžauskas was given the title of honorary canon.
Having travelled to Warsaw for treatment, the Rev. Silvestras Gimžauskas died on 27 October 1897. He was buried in Poland. The precise location of his grave is unknown.
To commemorate this distinguished figure, a parish house was built in Kirdeikiai and in 1936 was named in honour of the priest. The house in which he was born was moved to the Open-Air Museum of Lithuania in Rumšiškės (Kaišiadorys District) in 1974. At the site of the former homestead, a roofed chapel-pole was erected in 1988 (sculptor Stasys Karanauskas), and a street was renamed in his honour.