The question of whether or not genuine human progress is possible, or desirable, lies at the heart of many a radical post-Enlightenment philosophical project. More pessimistic philosophers have, unsurprisingly, doubted it. Arthur Schopenhauer, cast baleful suspicion on the idea. Danish Existentialist Soren Kierkegaard thought of collective progress toward a more enlightened state an unlikely prospect. One modern critic of progress, pessimistic English philosopher John Gray, writes in his book Straw Dogs that "the pursuit of progress" is an idealist illusion ending in "mass murder." (Gray has been unimpressed by Steven Pinker's optimistic arguments in The Better Angels of Our Nature.)
These skeptics of progress all in some way write in response to the towering 19th century figure G.W.F. Hegel, the German logician and philosopher of history, politics, and phenomenology whose systematic thinking provided Karl Marx with the basis of his dialectical materialism. Hegel saw the mass murder brought about by massive political and economic change in his revolutionary and imperial age, but in his estimation, such man-made disasters were necessary occurrences, the "slaughter bench of history," as he famously wrote in the Philosophy of History.
This suggests a very brutal view, and yet Hegel believed overall that "Reason is the Sovereign of the World; that the history of the world therefore, presents us with a rational process." For Hegel, the individual personality was not important, only collective entities: peoples, states, empires. These moved against each other according to a metaphysical reasoning process working through history which Hegel called the dialectic. In his animated School of Life video above, Alain de Botton describes the dialectic in the terms we usually use---thesis, antithesis, synthesis---though Hegel himself did not exactly formulate the principle this way.
This is the common shorthand way of understanding how Hegel's nonlinear explanation of history works: "the world makes progress," summarizes de Botton, "by lurching from one extreme to the other, as it seeks to overcompensate for a previous mistake, and generally requires three moves before the right balance on any issue can be found." One particularly bloody example is the terror of the French Revolution as an extreme corrective for the monarchy's oppression. This gave way to the antithesis, the brutal autocratic empire of Napoleon in another extreme swing. Only decades later could these be reconciled in modern French civil society.
In our own time, we have encountered the progressive ideas of Hegel not only through Marx, but through the work of Martin Luther King, Jr., who studied Hegel as a graduate student at Harvard and Boston University and found much inspiration in the Philosophy of History. Though critical of Hegel's idealism, which, "tended to swallow up the many in the one," King discovered important first principles there as well: "His analysis of the dialectical process, in spite of its shortcomings, helped me to see that growth comes through struggle."
We endlessly quote King's statement, "the arc of history is long, but it bends toward justice," but we forget his corresponding emphasis on the necessity of struggle to achieve the goal. As Hegel theorized, says de Botton above, "the dark moments aren't the end, they are a challenging but in some ways necessary part... imminently compatible with events broadly moving forward in the right direction." King found his own historical synthesis in the principle of nonviolent resistance, which "seeks to reconcile the truths of two opposites," he wrote in 1954's Stride Toward Freedom, "acquiescence and violence." Nonviolent resistance is not passive compliance, but neither is it intentional aggression.
Hegel and his socially influential students like Martin Luther King and John Dewey have generally operated on the basis of some faith---in reason, divine justice, or secular humanism. There are much harsher, more pessimistic ways of viewing history than as a swinging pendulum moving toward some greater end. Pessimistic thinkers may be more rigorously honest about the staggering moral challenge posed by increasingly efficient means of mass killing and the perpetuation of ideologies that commit it. Yet it is partly through the influence of Hegel that modern social movements have embraced the necessity of struggle and believed progress was possible, even inevitable, when it seemed least likely to occur.