Also eight ministers, except two, lost the polls along with party chief Subhash Barala.
The rival Congress won 31 seats, while the less than year old Jannayak Janata Party (JJP), which broke away from the state's once major regional Indian National Lok Dal (INLD) owing to family disputes, won 10 seats.
The INLD won a lone seat against 19 in 2014 Assembly election. Seven Independents and one each of the INLD and the Haryana Lokhit Party have also won.
While Khattar retained his Karnal seat with a comfortable margin, two-time Chief Minister and Congress leader Bhupinder Singh Hooda too managed to win in his bastion of Garhi Sampla-Kiloi in Rohtak district with a good lead.
Khattar won by 45,188 votes over his nearest rival, Tarlochan Singh of the Congress.
BJP's outgoing ministers - Captain Abhimanyu, O.P. Dhankar, Ram Bilas Sharma, Kavita Jain, Krishan Lal Panwar, Manish Grover, Karan Dev Kamboj and Krishan Kumar Bedi - all ended up with humiliating defeats.
But Public Health and Engineering Minister Banwari Lal won the Bawal seat and Health Minister Anil Vij retained his seat from Ambala Cantonment.
Two sitting ministers, Rao Narbir Singh and Vipul Goel, were denied tickets and fresh faces were brought in. Both lost the elctions.
Sitting Congress legislator Kuldeep Bishnoi, whose hotel worth Rs 150 crore in a prime location of Gurugram, has been attached by the Income Tax department, retained his stronghold of Adampur by defeating his nearest rival and TikTok celebrity Sonali Phogat of the BJP by 29,471 votes.
JJP chief Dushyant Chautala and his mother Naina Chautala won the Uchana Kalan and Badhra seats, respectively.
Outgoing Finance Minister Captain Abhimanyu, who lost to JJP candidate and BJP rebel Ram Kumar Gautam in the Narnaud constituency in Hisar district by a margin of 12,029 votes, admitted his defeat during counting.
As it was becoming clear that the BJP would not form the government on its own, its state unit chief Subhash Barala also lost the Tohana Assembly seat in Fatehabad district by 52,302 votes to JJP's Devender Singh Babli.
BJP candidate and former Indian hockey captain Sandeep Singh defeated Congress' Mandeep Chatha in Pehowa. This was the BJP's first win in the constituency.
While BJP candidate and international wrestler Yogeshwar Dutt faced defeat, another of its candidates, wrestler Babita Phogat, emerged third in the Dadri constituency in Bhiwani.
Another prominent defeat for the BJP is its sitting legislator Prem Lata, 59, wife of former Union minister Birender Singh, who was defeated by JJP chief Dushyant Chautala, great-grandson of former Deputy Prime Minister Devi Lal, by 47,452 votes.
Congress' national spokesperson Randeep Singh Surjewala, who lost by a mere 1,246 votes to BJP's Leela Ram, conceded defeat from Kaithal.
Surjewala, a two-time MLA, was seeking his third term from Kaithal, once a known stronghold of the Surjewala family.
Congress leader Kiran Chaudhary, however, retained her Tosham seat by 18,059 votes.
The Congress candidate from Ferozepur Jhirka in Mewat, Mamman Khan, defeated BJP's Naseem Ahmed, an INLD defector by 37,004 votes. Ahmed had won this seat in the 2014 and 2009 Assembly elections.
BJP's Zakir Hussain, who was the sitting legislator in Nuh constituency, lost his seat to Congress' Aftab Ahmed as the latter won by a margin of 4,038 votes.
BJP's Pramod Vij and Mahipal Dhanda won Panipat City and Rural seats, respectively.
The JJP appears set to play kingmaker in the state. Dushyant Chautala said he has called a meeting of the party executive at 11 a.m. on Friday in Delhi to discuss the party's strategy.
JJP candidate Ram Niwas won from Narwana (reserved). He contested against BJP's Santosh Rai. JJP candidate Ram Karan won from Shahbad (reserved) by a margin of 37,127 votes against BJP's Krishan Kumar.
Another JJP candidate Amarjeet Dhanda has won from Julana beating the incumbent Parminder Singh Dhull, who switched to the BJP from the INLD.
Independent Nayan Pal Rawat won the elections by 16,429 votes by defeating Congress candidate Raghubir Tewatia.
INLD's Abhay Chautala won the party's lone seat of Ellenabad with 11,922 votes, defeating BJP candidate Pawan Beniwal.
Haryana Lokhit Party (HLP) President Gopal Kanda registered a win from Sirsa seat with a slender margin of 602 votes, defeating Independent Gokul Setia.
Kanda extended unconditional support to the BJP.
After winning his seat by 58,312 votes by defeating BJP candidate Satish Nandal, Hooda said the mandate is against the BJP government.
"The mandate is against the current government of Haryana, and all parties should come together to form a strong government whether it's JJP, BSP, INLD or independent candidates," Hooda told reporters.
( With inputs from IANS )