Identification and tracking of multiple objects are essential in many applications. As a key enabler of automatic ID technology, RFID has got widespread adoption with item-level tagging in everyday life. However, restricted to the computation capability of passive RFID systems, locating or tracking tags has always been a challenging task. Meanwhile, as a fundamental problem in the field of computer vision, object tracking in images has progressed to a remarkable state especially with the rapid development of deep learning in the past few years. To enable lightweight tracking of a specific target, researchers try to complement computer vision to existing RFID architecture and achieves fine granularity. However, such solution requires calibration of the cameras extrinsic parameters at each new setup, which is not convenient for usage. In this work, we propose Tagview, a pervasive identifying and tracking system that can work in various settings without repetitive calibration efforts. It addresses the challenge by skillfully deploying the RFID antenna and video camera at the identical position and devising a multi-target recognition schema with only the image-level trajectory information. We have implemented Tagview with commercial RFID and camera devices and evaluated it extensively. Experimental results show that our method can archive high accuracy and robustness.