It is well known that tumors can be “seen” by the immune system, but immune responses differ depending on the nature and context of the target antigen: tumor antigens that are self-proteins are generally weakly immunogenic due to pre-existing self-tolerance, whereas tumor antigens that are truly tumor-specific (viral and mutated proteins) are potentially highly immunogenic because the immune system has not been previously exposed to these antigens. Therefore, tumor-specific T cells that recognize these antigens as non-self should be able to eliminate cancer cells expressing such tumor-specific neo-antigens. We developed a spontaneous Tamoxifen-inducible cancer mouse model to investigate the fate of naïve tumor-specific T cells encountering a tumor-specific neo-antigen during the pre-malignant phase of tumor development. We found, contrary to our expectation, that tumor-specific T cells were rendered tolerant and unresponsive to the cancer cell as early as 8 days post-tumor initiation and displayed very similar phenotypic and functional characteristics compared to self-tolerant T cells. Interestingly, this unresponsiveness was not due to global tumor-induced immune suppression in the microenvironment but instead the result of continuous antigen encounter, as tumor-infiltrating control T cells specific for an antigen not expressed by the cancer cell remained functional in the tumor site. Transcriptional profiling uncovered the molecular program underlying this early cancer antigen-induced T cell dysfunction. Thus, an antigenic stimulus in a non-inflammatory context, whether by a self-antigen or tumor-specific antigen, induces a distinct state of “tolerance-specific” functional unresponsiveness. While many immunotherapies aim to reverse the immune suppression mediated by the tumor microenvironment, we demonstrate that there is another level of antigen-specific, cell-intrinsic dysfunction that must also be overcome.