The key that must be kept secret is referred to as a private key, to distinguish it from the secret key of symmetric-key cryptography. This key is owned by and defines the unique identity of a particular entity (call it a server entity). The public key, as the name implies, is public and is commonly known to any entity (call it a client entity) that wants to exchange confidential communications with the owner of the private key.
The nature of the algorithms used with these key pairs is such that any client entity that holds a copy of the public key can engage in confidential communications with the server entity that owns the corresponding private key, and these communications are also confidential from other holders of the same public key. This works because any message encrypted using the private key can only be decrypted using the public key, and any message encrypted using the public key can be decrypted using only the private key. If a server entity maintains the secrecy of its private key (that is, if the server entity is secure), any client entity with the corresponding public key can know the identity of the secure server entity that it communicates with.
In a PKI, to assure confidentiality from other clients that possess the same public key, any client entity maintains data privacy by initiating its own secret-key communications with the server entity that owns the corresponding private key. This is what makes a PKI possible.
With symmetric-key cryptography alone, any entities seeking to engage in confidential communications must find a secure way to share the same secret key. The problems of managing a data security infrastructure with this constraint are formidable. With asymmetric-key cryptography, as used in a PKI, only one server entity needs to own the private (and secret) key. Any client entity can have access to the same public key without compromising data security for any other clients that want to exchange data with this server entity. Thus, the public key requires no security. The essential security for public-key cryptography depends only on the private key, which must remain completely confidential to the server entity that owns it. However, if server security is compromised, there can be no assurance that the server entity who owns the private key is the proper owner of that key. So, security for the server is critical to all clients that try to access it.