Just looked it up, the simple definition is "any kind of literature that requires special effort to navigate and digest." I'm thinking Infinite Jest or Ulysses would be good examples of that.