In 1948, Rehavam Ze'evi was a platoon commander in the IDF. In 1964–1968, he served as Chief of the Department of Staff in the Israeli General Staff. In the late 1960s, Ze'evi formed the elite Sayeret Kharuv, an anti-terror battalion, at the time when IDF Chief of Staff Haim Bar-Lev had begun to focus manpower and budget on armoured tank units, resulting in huge cutbacks in infantry forces. Over the next five years he served as the Commander of the Central Military District. He retired in September 1973, but rejoined the army when the Yom Kippur War broke out on 6 October 1973. A close friend of IDF Chief of Staff David Elazar, he was appointed Special Assistant to the Chief of Staff. He retired with the rank of major-general in 1974.
Ze'evi, known for his concern for Israel's captured or missing soldiers, wore a military identity disc with their names on his neck.
It was revealed in 2004 that Ze'evi had been chosen to be responsible for the building of the armed forces of Singapore at a time when he was deputy head of the Operations Branch in IDF.
After a secret visit in 1965, he appointed then Colonel Yaakov (Jack) Elazari to be head of the team of secret military delegation, along with then Lieutenant Colonel Yehuda Golan and other IDF officers to train and build up Singapore Armed Forces. They were nicknamed "Mexicans" during their stay in Singapore.
Ze'evi was shot in the Dan Jerusalem Hotel, formerly called at the time, the Jerusalem Hyatt Hotel, in Mount Scopus on 17 October 2001 by four Palestinian gunmen.