A claim within the atheist community that attempts to justify non-belief in God is that atheism is the default position.
This mindset is often associated with the relatively new redefining of atheism as a "lack of belief in a god." Proponents will then argue that rocks, babies, and animals are all atheists. I've heard several atheistic students repeat this as well.
There are a few problems with this. First, if atheism is a "lack of a belief in a god or gods," then atheism cannot be true. It is not longer a propositional statement, such as "God does not exist." Instead, atheism becomes a property of an object or person. I could ask you whether you possess brown hair, and you could affirm or deny that attribute about yourself. Similarly, that's what atheism under this new definition has become. If you do not possess a belief in god, then you are an atheist.
This just leads to confusion and word games. What people want to know is how you answer the question, "Does God exist?" If yes, then you are a theist. If no, then an atheist. If not sure, then agnostic.
This redefining seems to be a response to the problem atheists have in proving their beliefs. You can't prove the universal negative of "God does not exist." How could you know you're right? It's simply question begging (assuming the thing you believe to be true in order to defend your case).
Another problem is that this line of thinking is the logical fallacy of a category error. A category error is assigning a value to something which can't have that value. A simple example is, "How tall is yellow?" Yellow cannot have the property category of height. That's a category error. Similarly, rocks, animals, and babies cannot hold abstract beliefs. That's not a property they are capable of having.
The closest one could come with this line of thought is that babies could be agnostic by default. They "don't know." However, this is assuming that babies have the ability to think about God's existence. Whenever children do start thinking about God, the question becomes whether they assume He exists or is uncertain. How can we know that? Survey parents?
Thankfully, some studies have been done on this already. Recent studies indicate that theism is actually the default belief of children. People naturally believe in God. (See also: "Born Believers: The Science of Children's Religious Belief" by Dr. Justin Barrett)
While I'm sure this is a developing field of study and many would want to resist the studies I've cited here, the main point is that there are hardly any good reasons to think that atheism is a default position. While some may want to continue justifying their beliefs, a fair minded person is going to see the faults with this line of argument.