Consider the conditional entropy and mutual information for the binary symmetric channel. The input source has alphabet \(X = \{0, 1\}\) and associated probabilities \(\left\{\frac{1}{2}, \frac{1}{2}\right\}\). The channel matrix is \(\begin{pmatrix} 1-p & p \\ p & 1-p \end{pmatrix}\) where p is the transition probability. Then the conditional entropy is given by :
\(-p\log(p) - (1-p)\log(1-p)\)
Conditional entropy \(H(Y \mid X) = \sum_{x} P(x)\, H(Y \mid X = x)\).
Given either input, the output is correct with probability \(1-p\) and flipped with probability p, so \(H(Y \mid X = x) = -p\log p - (1-p)\log(1-p)\) for both inputs.
With weights \(\frac{1}{2}\) and \(\frac{1}{2}\): \(H(Y \mid X) = -p\log(p) - (1-p)\log(1-p)\).
Check: \(1 + p\log p + (1-p)\log(1-p)\) is the mutual information \(I(X;Y) = H(Y) - H(Y \mid X)\), not the conditional entropy.
Hence, \(-p\log(p) - (1-p)\log(1-p)\).
Similar Questions
Suppose that from given statistics, it is known that meningitis causes stiff neck 50% of the time, that the proportion of persons having meningitis is \(\frac{1}{50000}\), and that the proportion of people having stiff neck is \(\frac{1}{20}\). Then the percentage of people who had meningitis and complain about stiff neck is :
Important Questions from Conditional Probability
A box contains 5 coins: 4 regular coins and 1 fake coin. When a regular coin is tossed, the probability 𝑃(ℎ𝑒𝑎𝑑) = 0.5 and for a fake coin, 𝑃(ℎ𝑒𝑎𝑑) = 1. You pick a coin at random and toss it twice, and get two heads. The probability that the coin you have chosen is the fake coin is _______. (rounded off to two decimal places)
- A clinic specializes in testing for a disease D. The result of the test can be either positive or negative.
A study revealed that if a person suffers from the disease D, the test result in that clinic comes out positive 80% of the time, and negative 20% of the time. If a person is not suffering from the disease D, the test comes out positive 10% of the time and negative 90% of the time. It is also known that among the general population, the disease D occurs in 30% of the individuals.
If a person tests positive for D in that clinic, the probability that he/she actually suffers from the disease D is __________ . (Rounded off to two decimal places) - Suppose an unbiased coin is tossed 6 times. Each coin toss is independent of all previous coin tosses. Let $E_1$ be the event that among the second, fourth, and sixth coin tosses, there are at least two heads. Let $E_2$ be the event that among the first, second, third, and fifth coin tosses, there are equal number of heads and tails.
The conditional probability P($E_1$ | $E_2$) is equal to ____________. (rounded off to one decimal place) - A box contains three apples and two oranges. Two fruits are removed randomly in succession. The probability that the first is an apple and the second is an orange is _____ .
- An experiment is performed by rolling an unbiased die once. Events A and B are defined as follows:
Event A: The outcome of the rolled die is a prime number.
Event B: The outcome of the rolled die is an odd number less than 4.
The conditional probability of A given B, denoted by P(A|B), is ______ (rounded off to one decimal place).
