From April 2017, 600,000 small firms will not have to pay business rates, while 250,000 will pay lower rates.
To fund the giveaway, the chancellor capped debt interest payments used by larger firms to cut their corporation tax bills, to 30% of earnings.
George Osborne also said corporation tax would be cut to 17% by 2020.
Mr Osborne said his business tax reforms were part of a "Budget for small business".
The chancellor announced he was permanently raising the threshold for small business rate relief from £6,000 to a maximum of £15,000 and increasing higher rate relief from £18,000 to £51,000.
He said that the 600,000 small businesses relieved of paying business rates from next April would see an annual saving of nearly £6,000.
Mike Cherry, policy director at the Federation of Small Businesses, said: "The chancellor has listened to our calls for the tax system to be made simpler for small businesses and the self-employed and taken important action on business rates."
The chancellor also announced a number of reforms designed to help small businesses cope with what he called "the great unfairness" they faced when trying to compete with some suppliers selling goods online.
Mr Osborne said the government would take measures to stop overseas retailers storing goods in the UK and then selling the goods online without paying VAT.
He also introduced two new tax-free allowances worth £1,000 a year for so-called micro-entrepreneurs: people who make money from occasional jobs or through renting out property they own.
